A Linear Programming Approach to Nonstationary Infinite-Horizon Markov Decision Processes

Nonstationary infinite-horizon Markov decision processes (MDPs) generalize the most well-studied class of sequential decision models in operations research, namely, that of stationaryMDPs, by relaxing the restrictive assumption that problem data do not change over time. Linearprogramming (LP) has been very successful in obtaining structural insights and devising solutionmethods for stationary MDPs. However, an LP approach for nonstationary MDPs is currentlymissing. This is because the LP formulation of a nonstationary infinite-horizon MDP includescountably infinite variables and constraints, and research on such infinite-dimensional LPs hastraditionally faced several hurdles. For instance, duality results may not hold; an extremepoint may not be a basic feasible solution; and in the context of a Simplex algorithm, a pivotoperation may require infinite data and computations, and a sequence of improving extremepoints need not converge in value to optimal. In this paper, we tackle these challenges andestablish (1) weak and strong duality, (2) complementary slackness, (3) a basic feasible solutioncharacterization of extreme points, (4) a one-to-one correspondence between extreme points anddeterministic Markovian policies, and (5) devise a Simplex algorithm for an infinite-dimensionalLP formulation of nonstationary infinite-horizon MDPs. Pivots in this Simplex algorithm usefinite data, perform finite computations, and generate a sequence of improving extreme pointsthat converges in value to optimal. Moreover, this sequence of extreme points gets arbitrarilyclose to the set of optimal extreme points. We also prove that decisions prescribed by theseextreme points are eventually exactly optimal in all states of the nonstationary infinite-horizonMDP in early periods.